Baguamarsh: An Immersive Narrative
Visualization for Conveying Subjective
Experience
Fei Jiang
1,2(&)
, Don Derek Haddad
2
, and Joseph Paradiso
2
1
Shanghai Academy of Fine Arts, Shanghai, China
fred_jf@hotmail.com
2
MIT Media Lab, Cambridge, USA
Abstract. As ubiquitous sensing becomes embedded in our everyday world,
we can easily obtain multimodal sensor data from our physical surroundings.
Objectivity is a core value in the visualization research community. However,
personal experiences are characterized not only by objective facts, but by per-
sonal emotions. In this paper, we explore immersive data visualization for
conveying personal subjective perception and experience by using multidi-
mensional data and multimedia. We introduce a framework to describe narrative
structures in immersive data visualization, and provide an example project
Baguamarsh as a proof of theme-based creative methods for designing corre-
lations between different information. We hope this study may offer outline
opportunities for future research in narrative visualization.
Keywords: Virtual reality
Narrative visualization Multidimentional data
Storytelling Bagua Book of Changes
1 Introduction
As ubiquitous sensing becomes embedded in our everyda y world, we can easily obtain
multimodal sensor data from our physical world, as personal health data, meteoro-
logical data, geographical data and etc. There are many approaches that have been
designed to present date for users to communicate information clearly and efciently.
Data visualizations are commonly presented quantitatively, as statistical graphi cs, plots,
charts and other tools [1]. Objectivity is a core value in the visualization research
community [2]. However, personal experiences are characterized not only by objective
facts, but by personal emotions [3]. How to convey personal subjective perception and
experience by using multidimensional data becomes both an opportunity and a chal-
lenge in visualization eld.
On the one hand, storytelling is an effective way of conveying information and
enhancing understanding [4], as narrators can leverage personal information (data,
photos, audio, etc.) and express their subjective ideas through narrative visualization.
On the other hand, as VR headset s have become affordable for individuals, virtual
reality becomes a user-friendly platform for reconstructing and representing digital
information.
© Springer Nature Switzerland AG 2020
M. Kurosu (Ed.): HCII 2020, LNCS 12181, pp. 596613, 2020.
https://doi.org/10.1007/978-3-030-49059-1_44
In this paper, our approach treats the data visualization as a narrative and uses
virtual reality as a platform to convey a narrators story. Firstly, we discuss related
work on visualization research. Secondly, we introduce a framework to describe nar-
rative structures in immersive data visualization. Finally, we provide an example
project Baguamarsh as a proof of concept. Each of these aspects plays a great role in
allowing the narrator to affect the observer emotionally and intellectually by conveying
a rich, signicant idea.
2 Related Work
2.1 Pre-research
Project Moments is a prototype system for saving and representing personal
moments [5] and presents an overview of a whole process of creating an immersive
data visualization, whi ch provide a good reference value for this study.
A wireless sensor network (deployed at Tidmarsh Wildlife Sanctuary in south-
eastern Massachusetts) [6], a nd Chain-API (a RESTful service providing sensor data) ,
provide a place instrumented with a dense senor network for eld study and an open-
data platform for collecting environmen tal data, hence provides an ideal setting for this
project.
2.2 Subjectivity in Narrative Visualization
As narrative visualizations combine patterns of communicative and exploratory
information visualization to convey intended stories [4], many communities have
commented on the importance of narrative in data visualization [9]. But according to
Tong et al. [7], comparing with the development in other elds, storytelling is a
relatively new subject in visualization. On one hand, even storytelling has become a
common topic of discussion in data visualization now, but most research commonly
represents objective facts [7], and there is only a little research on personal storytelling
[3]. On the other hand, there are some data artworks that express subjectivity as a
design goal, but they do not describe specic techniques and methodologies for cre-
ating subjective representation [3].
2.3 Immersive Data Visualization
Virtual reality not only has been shown to lead to better discovery in domains whose
primary dimensions are spatial [8], but also to enhance situation awareness [9] and
media richness [10]. On one hand, according to Donalek et al., as virtual reality can
maximize the intrinsic human pattern recognition skills [8], immersive data visual-
ization provides more intuitive data understanding than traditional desktop visual-
ization tools. On the other hand, the public has understood that virtual reality already
portended a new medium for almost two decades [11], and we have seen much research
Baguamarsh: An Immersive Narrative Visualization 597
on interactive experience, presence, immersion and interaction [1217], but not many
works that address immersive narrative visualization involve live sensor data.
3 Framework
We explore the use of immersive virtual reality platforms for narrative visualization to
convey subjective experience. To achieve our goal, we introduce a theme-based
framework (shown in Fig. 1) with which a user not only can convey personal sub-
jective perception and experience by using multimedia and multidimensional data, but
also can enhance multidimensional expression.
According to the commonly used pipeline for collecting data [3] and creating
information visualizations [18], we inject a theme-based method in workow to rep-
resent subjective personal narratives in an immersive environment. Firstly, a main
theme chosen by a narrator is the selection criteria for collecting information (data,
media, etc.). Secondly, according to the theme, the narrator can choose to simplify,
classify, con nect or reconstruct information materials. Thirdly, rhetorical strategy is
used to map all content for representing annotation, user interface, presence and
interactivity. Finally, perceptualization is integrated for building up an immersive and
interactive environment.
4 Project Introduction
In this section, we present a project Baguamarsh that exemplies the goals described
in last section. This project takes the idea of the unity between heaven and man of the
ancient Chinese philosophy I Ching (Book of Changes) [19], combines the Bagua
(Eight Trigrams) of the I Ching with multidimensional data and multimedia, and uses
an immersive interactive environment to present a novel form of narrative visualization.
The main theme of this project is the Unity between Heaven and Man, namely the
harmony of nature and human. It systematically expounds the relationship between
man and nature, that man must follow the laws of the universe, to respect and protect
nature, and to have an insight into the truth that harmony can produce all things.
According to the main theme, we designed this project under the form of a virtual
reality environment by integrating personal data, environmental data and media
materials.
An I Ching framework has been used in prior work to interpret and represent
wearable sensor data for a quantied self application [20], but not to our knowledge for
integrated personal and large-scale environments sensing.
598 F. Jiang et al.
5 Information Collection and Processing
The data used for this project is divided into user data and environmental data. The user
data is mainly from wearable devices, that is to say an Apple Watch 4 and a Muse
BioHarness in this project. These two kinds of devices are separately responsible for
collecting a narrators ECG and EEG. Environmental data is mainly obtained in two
ways: (1) a GoPro 360 camera is responsible for taking 360° panoramic photos and
recording ambisonic audio from user environmen ts (shown in Fig. 2), (2) Using the
open data interface to obtain environmental data through time and GPS information
encoded in the panoramic photo EXIF information.
Fig. 1. Theme-based framework for constructing immersive narrative visualization
Baguamarsh: An Immersive Narrative Visualization 599
After the narrator wearing the wearable devices enters the data collection area
(Tidmarsh), he/she rstly takes a panoram ic photo with the 360 camera, and then starts
recording a panoramic video of about 30 s in the same position (the video contains
ambisonic audio) (show n in Fig. 3). Dur ing the video recording process, he/she needs
to collect ECG and EEG data for about 30 s when the mobile phone is connected with
the wearable device. The narrator may take advantage of the features of the 360°
camera, as he/she doesnt need to focus or frame when shooting image material, so
he/she can collect other data at the same time, which greatly shortens the information
collection tim e. After actual operation, we nd that the information collection process
for one location usually takes no more than 2 min.
Fig. 2. Devices for information collecting
Fig. 3. Information collection in Tidmarsh
600 F. Jiang et al.
Information processing is mainly divided into three steps: information extraction,
data expansion and data archiving (shown in Fig. 4). (1) Information extraction: mainly
for the image materials captured by the 360° camera, which includes panoramic photos
and panoramic videos. The panorama photo is a JPG image le, and each le contains
EXIF information. We use a program written in Python to extract the time and GPS
information from each photo for later use in data expansion. The extraction process for
panoramic video le information is relatively time consuming. We import the
panoramic video into the post-processing software Premiere before extracting the 4-
channel ambisonic audio. In addition, we use a sound generation program written in
Python to convert the ECG signals collected in the previous period (Fig. 4, top) into 30-
s heartbeat sounds, which will be used in the later virtual reality interacti on. (2) Data
expansion: using the time and GPS information in the previous photos, we obtain the
24-h environmental information of the information collection day through an open data
interface. The open data platforms used in this project include the following: Google
Maps API, AccuWeather APIs, Chain API [21]. We use Python-written programs to
get locations and elevations from the Google Map API, get temperatures and winds
across the entire area from AccuWeather APIs, and get the pressure, relative humidity,
visible light, intensity of infrared and ultr aviolet light, soil temperature and humidity
from the Chain API in the area where the user is located at Tidmarsh. We put all of this
data together to generate a text le in JSON format. (3) Data archiving: We use the
desktop program written by Unity to package panoramic photos, ambisonic audio and
text les containing all metadata into an archive le, which is convenient for subse-
quent virtual reality program calls.
6 Content Mapping
The way we present content has the following characteristics: (1) Narrative visual-
ization will be presented in the VR environment reconstructed from 360 panoramic
photos, and users can switch between the real environment and special effects envi-
ronment. (2) By combining the physical properties of the phenomena described by the
data with commonly used visualization methods, we develop novel and user-friendly
modes of presentation. (3) Eight Trigrams (see below) will persist across the entire
visual representation, and the symbols corresponding to each kind of trigram will be
adopted as the basic visual elements of the environmental data points, which also
reect the connection between data and trigrams on the micro level and make the
observation more convenient, thereby improving annotation. (4) Most of the visual-
ization objects in this project are time series datasets in the unit of scenes. All data
points can not only display the visualization effects at different times and in different
trigrams, but also reect the relationship between personal data and environmental data,
thereby abstracting a visua l pattern of Unity between Heaven and Man to users.
Baguamarsh: An Immersive Narrative Visualization 601
Fig. 4. Information processing
602 F. Jiang et al.
Eight Trigrams are the basic concept of the Book of Changes and can represent the
dynamic and still states of all natural phenomena. Eight kinds of symbols used in the
Eight Trigram correspond to eight phenomena in nature (Heaven, Earth, Water, Fire,
Thunder, Wind, Mountain, Marsh), and their names are Qian, Kun, Kan, Li, Zhen,
Xun, Gen and Dui. Through the evolution of these eight natural phenomena, we
speculate on the laws of man and nature in an attempt to achieve a world view of
harmony between humanity and nature. The Eight Trigrams are divided into Earlier
Eight Trigrams and Later Eight Trigrams. What is used in the project is the Later Eight
Trigrams, which are said to have been made by Zhou Wenwang. In the Later Eight
Trigrams, the Trigram Zhen is the starting point and it is due east. According to the
clockwise direction order, there are in sequence: Trigram XunSoutheast; Trigram Li
South; Trigram KunSouthwest; Trigram DuiWest; Trigram QianNorthwest;
Trigram KanNorth; Trigram GenNortheast. The order of the Later Eight Trigrams
are: one of Kan, two of Kun, three of Zhen, four of Xun, ve of the center, six of Qian,
seven of Dui, eight of Gen, and nine of Li. The ancients used to draw maps with south
at the top and east in the left. We create a three-dimensional interactive menu based on
graphics of Eight Trigrams and the shape of the Fengshui Bagua mirror for the
interaction of the VR controller (shown in Fig. 5).
According to the eight phenomena descri bed in the Bagua, we divide the acquired
environmental data into eight categories that correspond to each of the Trigrams
(shown in Fig. 14) in the menu: (1) Trigram Qi an: atmospheric pressure (2) Trigram
Dui: soil moisture (3) Trigram Li: UV index (4) Trigram Zhen: cloud cover (5) Trigram
Xun: wind speed (6) Trigram Kan: humidity (7) Trigram Gen: elevation (8) Trigram
Kun: soil temperature. Users can interact with the data space through the Bagua menu:
the specic visualizations are as follows (Figs. 6, 7, 8, 9, 10, 11, 12 and 13):
Fig. 5. Bagua menu
Baguamarsh: An Immersive Narrative Visualization 603
Fig. 6. Trigram Qian atmospheric pressure (AP): the atmospheric pressure data is used to
control the volume of a ring around an object. The higher the pressure, the larger its volume.
Fig. 7. Trigram Dui soil moisture (SM): the soil moisture data is used to simulate the water on
the ground. The greater the moisture value, the wider the line of water.
Fig. 8. Trigram Li UV index (UV): the UV index represents twinkling lines in the sky. The
higher the value, the high their density.
Fig. 9. Trigram Zhen cloud cover (CC): the value of cloud cover is used to control the density
of clouds in the sky. Cloud density increases with the value of cloud cover.
604 F. Jiang et al.
Fig. 13. Trigram Kun soil temperature (ST): the soil temperature is used to control the color
tone of ground material. The higher the temperature, the warmer the color.
Fig. 10. Trigram Xun wind speed (WS): wind data is used to control the rotational speed of
visual elements. The higher its value, the faster the rotational speed.
Fig. 11. Trigram Kan humidity (Hu): humidity is used to control the density of visual
elements. The larger the value, the higher the density.
Fig. 12. Trigram Gen elevation (El): elevation data controls height differences between water
particles and other objects. The higher the value, the bigger the gap.
Baguamarsh: An Immersive Narrative Visualization 605
According to the I Ching, Trigram Qian corresponds to the bodys brain, and
Trigram Li corresponds to the heart. We build the EEG and ECG into a dynamic
model, then place them at the cen tre of the interactive menu. Both the EEG and ECG
data are displayed as line graphs around the model, and all environment data is shown
on the submenus attached to the interactive menu (shown in Fig. 5).
The green main menu can be rotated by the VR controller. Each Trigram contains
different kinds of data and different combinations of the Five Elements which are
designed for triggering visual effects.
Users can select the environmental data within 24 h of the day of the encounter
through the menu selection. We map the elements represented by each trigram to the
relationship between the ve elements. We superimpose the elements corresponding to
these attributes over the visual elements manifested by different data to re ect the
process of mutual promotion and restraint between the ve elements (shown in Fig. 15
and Fig. 16), reected by the Eight Trigrams. The calculation formulas for each Tri-
gram are as follows:
Qian: fAPðÞ¼AP þ EEG þ El UV ð1Þ
Dui: fSMðÞ¼SM þ ST UV ð2Þ
Li: fUVðÞ¼UV þ ECG þ CC þ WS Hu ð3Þ
Zhen: fCCðÞ¼CC þ Hu SM ð4Þ
Xun: fWSðÞ¼WS þ Hu AP ð5Þ
Kan: fHuðÞ¼Hu þ AP þ SM ST El ð6
Þ
Gen: fElðÞ¼El þ UV CC ð7Þ
Kun: fSTðÞ¼ST þ UV WS ð8Þ
606 F. Jiang et al.
Fig. 14. Data mapping of eight Trigrams
Fig. 15. The production and destruction relations in ve elements
Baguamarsh: An Immersive Narrative Visualization 607
Before we use the algorithm, there are very little differences between each time
point, so users can hardly feel the interaction in VR. By using the calculation formulas
of the mutual promotion and restraint between different data, the visual effect changes
dramatically, which can provide the user a better interactive experience. When a user
chooses a different Trigram, the lines between each element will be changed. The green
line represents production and the red line represents destruction (shown in Fig. 16).
Fig. 16. The algorithm of relations for data interaction
608 F. Jiang et al.
7 Immersive Interaction
For highly immersive visua lization purposes, a rst-person view is appropriate (shown
in Fig. 18). The observer will experience both visual and audio cues. This project is
currently available as a prototype application for Oculus Go, which has a relatively
high performance/price ratio on the market. This VR headset does not need to be
connected to a computer, which makes a better user experience. In terms of software
production, we use Unity3D to import the archive les that were previously packaged
by the desktop program into the VR main program in batches, and generate an Android
program that supports Oculus Go. Based on the characteristics of the device, we design
a VR interactive mode according to the mode chosen in the menu.
When a user is in the VR environment, he/she can not only walk around, look
around, hear 3D background sound, selec t scenes, and hide or move visual objects, but
also can select different trigrams to explore different visualization and auralization
results. For example, when a user chooses Trigram Li, it will begin to rain Trigrams
(shown in Fig. 19). He/she can hear the sound of heartbeats and a UV Index data value
is loaded for the interactive menu. The users can switch between the real environment
and the data environment (shown in Fig. 21), or change the time to see what would
happen to the environment then. He/she can even use the controller cursor to select a
3D object to change its appearance (shown in Fig. 20) based on the value of Trigram
that he/she chose.
Fig. 17. Different options presents different hints (Color gure online)
Baguamarsh: An Immersive Narrative Visualization 609
Fig. 18. First-person view
Fig. 19. Trigram rain for notication
610 F. Jiang et al.
Fig. 21. Sky of data environment
Fig. 20. Change the appearance of a tree
Baguamarsh: An Immersive Narrative Visualization 611
8 Conclusion
In this paper, we introduced a framework and an ongoing project to describe how to
design an immersive narrative visualization for conveying subjective experience. We
describe the whole process of project Baguamarsh for deliberately expressing subjec-
tivity during information collection, processing, interaction and presentation. These
preliminary studies show us the rst insights into the potential of immersive data
visualization as a subjective storytelling platform and prompt a discussion for future
research on conveying subjective experience in personal visual storytelling. More
visuals and video from this project can be view ed at https://www.media.mit.ed u/
projects/baguamarsh/overview/.
Acknowledgments. This study is supported by The Responsive Environments Group at
The MIT Media Lab. This group mainly explores how sensor networks augment and mediate
human experience, interaction, and perception, while developing new sensing modalities and
enabling technologies that create new forms of interactive experience and expression.
References
1. Friendly, M.: Milestones in the history of thematic cartography, statistical graphics, and data
visualization (2008)
2. Jorgenson, L., Kritz, R., Mones-Hattal, B., Rogowitz, B., Fraccia, D.F.: Is visualization
struggling under the myth of objectivity? In: Proceedings of IEEE Visualization (1995)
3. Thudt, A., Perin, C., Willett, W., Carpendale, S.: Subjectivity in personal storytelling with
visualization. Inf. Design J. 23(1), 4864 (2017)
4. Segel, E., Heer, J.: Narrative visualization: telling stories with data. IEEE TVCG 16(6),
11391148 (2010)
5. Jiang, F.: A prototype system for saving and representing personal moments. In: Marcus, A.,
Wang, W. (eds.) HCII 2019. LNCS, vol. 11585, pp. 314322. Springer, Cham (2019).
https://doi.org/10.1007/978-3-030-23538-3_24
6. Mayton, B., et al.: The networked sensory landscape: capturing and experiencing ecological
change across scales. Presence Teleoper. Virtual Environ. 26(2), 182209 (2017)
7. Tong, C., et al.: Storytelling and visualization: an extended survey. Information 9, 65 (2018)
8. Donalek, C., et al.: Immersive and collaborative data visualization using virtual reality
platforms. In: 2014 IEEE International Conference on Big Data (2014)
9. Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Hum. Factors:
J. Hum. Factors Ergon. Soc. 37(1), 3264 (1995)
10. Klein, G., Militello, L.: Some guidelines for conducting a cognitive task analysis. Adv. Hum.
Perform. Cognit. Eng. Res. 1, 161199 (2001)
11. Bates, J.: Virtual reality, art, and entertainment. J. Teleoper. Virtual Environ. 1(1), 133138
(1991)
12. Jennett, C., et al.: Measuring and dening the experience of immersion in games. Int.
J. Hum.-Comput. Stud. 66, 641661 (2008)
13. Usoh, M., Catena, E., Arman, S., Slater, M.: Using presence questionnaires in reality.
Teleoper. Virtual Environ. 9(5), 497503 (2000)
612 F. Jiang et al.
14. Garau, M., Slater, M., Vinayagamoorthy, V., Brogni, A., Steed, A., Sasse, M.A.: The impact
of avatar realism and eye gaze control on perceived quality of communication in a shared
immersive virtual environment. In: Proceedings of the 2003 Conference on Human Factors
in Computing Systems, CHI, Ft. Lauderdale, Florida, USA (2003)
15. Heidicker, P., Langbehn, E., Steinicke, F.: Inuence of avatar appearance on presence in
social VR. In: 2017 IEEE Symposium on 3D User Interfaces (3DUI). IEEE (2017)
16. Schubert, T., Friedmann, F., Regenbrecht, H.: The experience of presence: factor analytic
insights. Teleoper. Virtual Environ. 10(3), 266281 (2001)
17. Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: a presence
questionnaire. Teleoper. Virtual Environ. 7(3), 225240 (1998)
18. Card, M.: Readings in Information Visualization: Using Vision to Think. Morgan
Kaufmann, Burlington (1999)
19. Wilhelm, H., Baynes, C.F., Jung, C.G.: The I Ching or Book of Changes. Princeton
University Press, Princeton (1997)
20. Yano, K.: AI for taking on the challenges of an unpredictable era. Hitachi Rev. 65,3539
(2016)
21. Russell, S., Paradiso, J.A.: Hypermedia APIs for sensor data: a pragmatic approach to the
web of things. In: Proceedings of the 11th International Conference on Mobile and
Ubiquitous Systems: Computing, Networking and Services (2014)
Baguamarsh: An Immersive Narrative Visualization 613